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Effectiveness of Fully Online Courses for College Students: 
Response to a Department of Education Meta-Analysis 



SUMMARY: Proponents of postsecondary online education were 
recently buoyed by a meta-analysis sponsored by the U.S. 
Department of Education suggesting that, in many cases, student 
learning outcomes in online courses are superior to those in 
traditional face-to-face courses. This finding does not hold, 
however, for the studies included in the meta-analysis that pertain 
to fully online, semester-length college courses; among these 
studies, there is no trend in favor of the online course mode. What 
is more, these studies consider courses that were taken by 
relatively well-prepared university students, so their results may 
not generalize to traditionally underserved populations. Therefore, 
while advocates argue that online learning is a promising means to 
increase access to college and to improve student progression 
through higher education programs, the Department of Education 
report does not present evidence that fully online delivery 
produces superior learning ou tcomes for typical college courses, 
particularly among low-income and academically underprepared 
students. Indeed some evidence beyond the meta-analysis suggests 
that, without additional supports, online learning may even 
undercut progression among low-income and academically 
underprepared students. 



Introduction and Background 

Over the past decade, online learning has become an increasingly popular option among 
postsecondary students. Yet the higher education community still regards fully online 
courses with some ambivalence, perhaps due to the mixed results of a large (if not 
necessarily rigorous) body of research literature. On the one hand, research suggests that 
students who complete online courses leam as much as those in face-to-face instruction, 
earn equivalent grades, and are equally satisfied (e.g., see Jahng, Krug, & Zhang, 2007; 
Phipps & Merisotis, 1999; Sitzmann, Kraiger, Stewart, & Wisher, 2006; Zhao, Lei, Yan, 
Lai, & Tan, 2005). On the other hand, online students are less likely to complete their 
courses (Beatty-Guenter, 2003; Carr, 2000; Chambers, 2002; Moore, Bartkovich, Fetzner, 
& Ison, 2003). 

Skeptics of online learning raise concerns about the quality of online coursework. Some 
note that rather than developing approaches to teaching that would take advantage of the 
capabilities of computer-mediated distance education, instructors in many cases simply 
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transfer their in-class pedagogy to an online format (see Cox, 2005). Others suggest that 
student-teacher and student-student interactions are often limited (Bambara, Harbour, 
Davies, & Athey, 2009). These practices may contribute to low online course completion 
rates. Institutions harbor particular concern about online course performance among 
underprepared or traditionally underserved students, who are already at risk for course 
withdrawal and failure. 

Advocates of online learning, in contrast, argue that technology-enhanced education can 
lead to superior learning outcomes, and that higher online dropout rates are due not to the 
medium per se but rather to the characteristics of students who choose online courses 
(see, e.g., Howell, Laws, & Lindsay, 2004). Advocates are also particularly optimistic 
about the potential of fully online coursework to promote greater access to college by 
reducing the cost and time of commuting and, in the case of asynchronous approaches, by 
allowing students to study on a schedule that is optimal for them. Indeed, this goal of 
improved access is one of the top drivers of institutional decision-making regarding 
increases in distance education offerings (Parsad & Lewis, 2008). 

Recently, proponents of postsecondary online education were buoyed by a meta-analysis 
commissioned by the U.S. Department of Education (2009) which concluded that, among 
the studies considered, student learning outcomes in hybrid-online and fully online 
courses were equal to or better than those in traditional face-to-face courses. This 
conclusion included the caveat, however, that the positive effect for online learning 
outcomes was much stronger when contrasting hybrid-online to face-to-face courses than 
when contrasting fully online to face-to-face courses. In addition, the positive effect was 
much stronger when the hybrid-online course incorporated additional materials or time 
on task which was not included in the face-to-face course. Ignoring these subtler 
implications, popular media discussions of the findings (e.g., Lohr, 2009; Lamb, 2009; 
Stern, 2009) focused on the report’s seemingly clear-cut generalization that “on average, 
students in online learning conditions performed better than those receiving face-to-face 
instruction” (U.S. Department of Education, Office of Planning, Evaluation, and Policy 
Development, 2009, p. ix). This interpretation has also extended into the discourse of the 
higher education community. For example, higher-education experts participating in an 
online panel for The New York Times cited the meta-analysis as showing that students in 
online courses typically have better outcomes than those in face-to-face courses 
(“College degrees without going to class,” 2010). In this paper, we argue that such an 
interpretation is not warranted when considering fully online courses in the typical 
postsecondary setting. We also discuss implications of the studies for student access and 
progression among traditionally underserved populations. 
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Scope and Relevance of the Meta-Analysis 



In contrast to previous reviews and meta-analyses that included studies of widely varying 
quality, the Department of Education report attempts to update and improve our 
understanding of online learning effectiveness by focusing on only rigorous research: 
random-assignment or quasi-experimental studies that compare learning outcomes 
between online and face-to-face courses. The meta-analysis includes both fully online 
and hybrid courses in its definition of “online courses.” However, for institutions that aim 
to increase student access, fully online course offerings are a much more relevant 
concern, given that most hybrid courses require students to spend a substantial proportion 
of time on campus. For example, of the 23 hybrid courses that were examined in studies 
included in the meta-analysis, 20 required the students to physically attend class for the 
same amount of time that students in a face-to-face course would attend; the online 
portions of these courses were either in on-campus computer labs or were completed in 
addition to regular classroom time. Scaling up such hybrid course offerings is unlikely to 
improve access for students who have work, family, or transportation barriers to 
attending a physical classroom at specified times. 

In keeping with the notion of improved student access as a strongly emphasized rationale 
for online learning, we first narrowed our focus to the 28 studies included in the 
Department of Education meta-analysis that compared fully online courses to face-to-face 
courses. Unfortunately, the majority of these studies are not relevant to the context of 
online college coursework for one of two reasons discussed more fully below: (1) 
conditions are unrepresentative of typical college courses, or (2) target populations are 
dissimilar to college students. 

First, over half of the 28 studies on fully online learning concerned not a semester-length 
course but rather a short educational intervention on a discrete and specific topic, with an 
intervention time as short as 15 minutes. Moreover, some researchers who conducted the 
studies noted that they chose topics for the intervention that were particularly well-suited 
to the online context, such as how to use an Internet search engine. These studies, in 
general, may demonstrate that students can learn information on a specific topic from a 
computer as readily as they can a human, but the studies cannot address the more 
challenging issues inherent in maintaining student attention, learning, motivation, and 
persistence over a course of several months. 1 Given that many college students do not 
complete their online courses, student retention across the semester is a particularly 
important issue. As a result, these studies are minimally helpful to college administrators 
who are contemplating the potential costs and benefits of expanding semester-length 
online course offerings. 



1 Some practitioners question the utility of the Carnegie system of awarding credits on the basis of “seat 
time” in semester-length courses; they suggest that online learning could help convert instruction and 
learning into assemblages of short modules, with credits based on mastery of specific skills and topic areas. 
While this is an interesting frontier for further discussion and exploration, it does not now (nor will it in the 
near future) represent a widespread phenomenon at postsecondary institutions. Accordingly, we feel it 
makes most sense to focus on postsecondary courses as they are now typically structured. 
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Second, the studies were conducted across widely varying target populations, including 
primary school students and professionals outside of the college setting. When 
considering only those studies conducted with undergraduate or graduate students in 
semester-long online courses, the set of 28 studies is reduced to 7. Below, we discuss 
these seven studies in more detail. 



Comparison of Student Learning Outcomes in the Seven Relevant Studies 

In each of the seven studies of fully online semester-length college courses included in 
the meta-analysis, the courses were asynchronous such that students could log on and 
view lectures or other course materials at any time, although some required periods of 
synchronous chat. In all studies, the lectures, materials, learning modules, quizzes, and 
tests presented in the online and face-to-face classrooms were reasonably equivalent. 

• Caldwell (2006) examined an introductory computer science class (focused on the 
programming language C++) at a historically Black state university. Students 
enrolled in the class were randomly assigned to one of three course modes: face- 
to-face (face-to-face lecture and labs, no web materials), web-assisted (lecture and 
course materials online, face-to-face lab), and online (all materials and lab 
assignments online), with 20 students in each group. The online group’s only 
communication with the instructor was via email, and the only communication 
with other students was through voluntary means such as chat or discussion 
boards. Across the course of the semester, no students from any group withdrew 
from the course. Six outcome measures were examined, including two multiple- 
choice midterm exams, three programming assignments, and a “proficiency” final 
exam. There were no significant differences between the groups on any of these 
outcomes. 

• Cavus and Ibrahim (2007) focused on a Java programming course at a private 
international university. Students enrolled in the course were randomly assigned 
to one of three course modes (face-to-face, online with standard collaboration 
tools, online with advanced collaboration tools), with 18 students in each mode. 
Both online courses included web-based course notes and quizzes, as well as 
voluntary chat and discussion forums. Students using the “standard” collaboration 
tool worked jointly with other students on programming code, then ran the 
programs on their own PCs. In addition, the “advanced” tool allowed students to 
run their programming projects online and to automatically share their outputs 
with other students and the instructor. Each online course met synchronously for 
two hours a week using the relevant collaborative tool, and online students also 
had the option of using the tools more often (although the extent to which they did 
so was not stated). Face-to-face students had no access to either online tool, and it 
is unclear whether other collaborative methods were built into the face-to-face 
course; it is also unclear whether the face-to-face students were taught in a lecture 
or a computer laboratory setting. Student withdrawal rates were not mentioned. 
The advanced-collaboration online course significantly outperformed both the 
standard-collaboration online and face-to-face courses on the midterm and final 
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